Computer-implemented method for detecting three-dimensional surface geometries

ABSTRACT

In a computer-implemented method for detecting three-dimensional surface geometries of real intraoral structures, two-dimensional images of intraoral structures are converted into three-dimensional virtual surface geometries. Pixels of the two-dimensional images are each partitioned into segments and each segment is assigned a label. Each segment of a two-dimensional image corresponds to at least one type of real intraoral structure. There are at least two types of segment; each type of segment has a different label and at least one of the labels represents teeth and tooth-like structures.

BACKGROUND OF THE INVENTION Field of the Invention

The invention relates to a computer-implemented method for detectingthree-dimensional surface geometries of real intraoral structures, inwhich two-dimensional images of intraoral structures are converted intovirtual three-dimensional surface geometries.

Description of the Related Art

Particularly in the field of dental treatments, there are many systemsfor optically detecting the three-dimensional surface geometry ofobjects. These are used, for example, when producing prostheses, crowns,inlays, or the like, are used for providing assistance when monitoringorthodontic treatments and/or providing general assistance in theobservation and detection of intraoral structures. The large advantageof such optical systems lies in the fact that on the one hand, they areneither invasive, nor uncomfortable, like the dental impressionsfrequently used in conventional dentistry, and they also do not presenta potential hazard to the patient as can be the case, for example, withradiation-based methods such as X-rays. On the other hand, afterdetection, the data are available in electronic form and can be easilystored, for example for later comparisons, or also transmitted, forexample from a dentist to a dental laboratory.

One of the essential tasks involved in improving methods for opticaldetection of surface geometries is improving the precision of thedetected surface geometry and the speed of the detection. For thispurpose, it is primarily desirable to achieve a correct and rapidmatching between the individual two-dimensional, 2.5-dimensional, and/orthree-dimensional images and/or segments.

SUMMARY OF THE INVENTION

The object of the invention, therefore, is to improve the processing oftwo-dimensional images for producing virtual three-dimensional surfacegeometries.

This object is attained according to the invention by means of acomputer-implemented method with the features of claim 1.

As defined by the invention, an image can be a single two-dimensionalshot or can be two or more two-dimensional shots that are combined intoone image.

The usual sequence in optical 3D scanning is that first, two-dimensionalimages are recorded, then based on these images, relief-like depthinformation is obtained, which is considered to be 2.5D, and then atleast two 2.5D data sets are combined to form virtual 3D objects. Thelast step usually requires at least one pair-by-pair intersection of the2.5D data sets. A proven method for converting 2D data into 3D data,which can also be used with the invention, is known, for example, fromU.S. Pat. No. 9,544,577 B2.

In general, with scanners that are not positioned in a constrainedspatial relationship to the object that is to be scanned, it iscustomary and necessary to determine the camera movement betweenindividual images. In this case, it is possible to determine either themovement of the camera in the literal sense or merely the mathematicaltranslation (translation vector) and rotation (rotation matrix) that arenecessary in order to integrate the new depth information into theexisting virtual three-dimensional surface geometries (the translationvector and the rotation matrix yield the translation matrix). Often, theterms “camera movement” and “translation matrix” are usedinterchangeably since they are equivalent for the data processing.

Stated very simply, for each 2.5D data set, a determination is made asto the position that the sensor was in relative to the intraoralstructure recorded and as to the viewing angle at which the underlyingtwo-dimensional image was recorded. The stringing together of sensorpositions or information about how a sensor has moved from one image asubsequent image, together with the depth information, yields thevirtual three-dimensional surface geometry. Thus for each region of thevirtual surface geometry, it is also possible to say how often it was“seen” by a sensor.

If labels that have been assigned according to the invention are alsoincluded in addition to the information of how often a particular regionof the virtual three-dimensional surface geometry has been “seen,” it isalso possible to glean information about how often a particular labelhas been “seen” in a particular region.

The fact that a region is “seen” is comparable to human sight in thiscontext and means that a sensor has detected an image of a structure andthat the detected image has been at least partially interpreted. Theinterpretation can include both the segmenting and also thedetermination of depth information.

According to a particularly preferred implementation of the invention,the labeling takes place by means of an artificial intelligence, theartificial intelligence having been trained to detect at least teeth indigital images. The artificial intelligence is preferably a neuralnetwork, in particular a convolutional neural network.

Before the generation of the three-dimensional surface geometries, thepixels, in addition to their already existing information, areautomatically classified into at least two different classes and areprovided with corresponding labels, with one of the labels representingteeth and tooth-like structures.

The information already present in a pixel can, for example, be thebrightness of the pixel and the coordinates of the pixel in the image.

As defined by the invention, a pixel can also be representative of aplurality of pixels, for example if the two-dimensional image has beencompressed before the segmenting.

It is possible that only one kind of intraoral structure, for exampleonly a tooth or only the gums, is depicted in an image. In this case,the image in the sense of the invention is partitioned into only onesegment and all of the pixels of the image are allocated to one segmentand provided with a label.

In this context, “teeth and tooth-like structures” can include anythingthat visually corresponds to a tooth. This can, for example, include,but is not limited to, structures from the following group: teeth,crowns, implants, inlays, fillings, bridges, and the like.

So that a segment represents teeth and tooth-like structures, in thesimplest implementation, the teeth and tooth-like structures can bedetected and, for example, classified as “tooth.”

But it is also possible, for example, to detect and classify all of theregions that are not teeth. The remaining label “remainder” in this caselikewise represents the teeth.

The procedure of having the artificial intelligence draw a distinctionbetween “tooth” and “not tooth” already on the two-dimensional leveldoes indeed constitute an additional step, which is why the personskilled in the art would expect for this to slow down the overallmethod, but the procedure results in such a plurality of advantages thatthe “lost” time can be made up again in the subsequent processing.

The time saved can be primarily explained by the smaller number oferroneous matchings that then have to be discarded. Matchings arebasically pairings between two or more components of at least twotwo-dimensional images about which a spatial relationship is known ordetermined. Depth information can be obtained from a two-dimensionalmatching with a known or determined spatial relationship. The componentsof the two-dimensional images in this case are in direct relation to thereal objects that are depicted in the images. It is important here thateach component that is used to obtain depth information can be detectedor determined as uniquely as possible. In other words: a high degree ofcertainty that two paired (or matched) elements in two two-dimensionalimages show the same region on the same object is decisive for thequality of the overall scanning procedure. This certainty increases iferroneous matchings can be ruled out. The components of thetwo-dimensional images can be one or more pixels in size.

If the two-dimensional images are classified as is provided according tothe invention and if regions and/or pixels of the images are labeled inaccordance with classifications, then matchings can be limited so thatthey only take place within the same label. This does not completelyrule out erroneous matchings, but it can avoid a large percentage of theerroneous matchings. Thus for example, teeth in one image and gums inanother image can no longer be inadvertently matched to each other.

It is preferable if the labels assigned in the two-dimensionalimages/regions are also present in subsequent data levels, i.e. in theform of 2.5D information and 3D information.

If the 2.5D information has segments, then this can simplify thecombining of 2.5D data sets to form 3D surfaces since here as well, itis necessary to search for reference points, and limiting the possiblereference points to those reference points that have the same labelsminimizes error sources and thus accelerates the procedure.

If the labels are retained into the three-dimensional data level, thenthis can be used for other steps such as a coloring of the 3D model. Itis thus possible, for example, for all surfaces with the class “tooth”to be colored white and for all of the remaining surfaces to be coloredpink. In a preferred modification of the invention, it is thus provided,for example, that the segments are allocated colors and that regions ofthe generated three-dimensional surface geometry are colored inaccordance with the labels of the corresponding segments of the digitalimages.

According to the invention, the segments correspond to types ofintraoral structures. According to a preferred modification of theinvention, the types of structures can, for example, be

-   -   soft tissue (such as the tongue, cheeks, and the like),    -   hard tissue (such as the gums or palate),    -   tooth-like structures (such as teeth or dentures that are        modeled on teeth),    -   artificial structures (such as parts of braces, dentures, which        are recognizable as such, parts of reconstructive preparations        such as abutments or the like),    -   transitions between individual types of structures, or    -   anything that cannot be identified (e.g. background or teeth too        damaged by cavities).

Depending on how the neural network has been previously trained, it ispossible to identify all of the above-mentioned types or also onlyselected types. All of the unidentified types can then be combined, forexample, into another type called “remainder.” A segment can alsocomprise several types of intraoral structures, for example “gums andsoft tissue” or “teeth, tooth-like structures, and artificialstructures.” The number of types of different segments resulting fromthe segmenting in this case affects both the complexity of the trainingof the neural network and the complexity (but also the precision) of thesubsequent scanning procedure.

Through the assignment of other labels, it is possible to detect moresurfaces, thus further accelerating and improving the method. Theremaining regions can, for example, be the background, but can also betransition regions in which the artificial intelligence was unable toassign any of the other labels.

In reality, it has turned out that it is particularly advantageous ifthe segments “tooth,” “gums,” “transition,” and “remainder” areidentified. In this case, for example, the segments “tooth,” “gums,” and“transition” can be identified and labeled and all of the unassignedregions of the images are automatically labeled as “remainder.”

Naturally, “gums” and “background,” for example, could also beidentified and everything else could be labeled as “tooth.” Whichpriorities are set here in the training of the neural network depends onthe user and has no influence, or only a negligible one, on thefunctionality of the invention.

In another preferred modification of the invention, in addition to therespective class, the labels include additional information. Thisinformation can, for example, be color information that has resulted inthe assessment “tooth” or “gums.”

In a preferred modification of the invention, in addition to the type ofintraoral structure, the labels also include a probability thatindicates how probable it is that a correct label has been assigned. Inanother preferred modification, this probability can flow into aweighting of the three-dimensional virtual surface geometry.

If the 3D information is stored, for example, in a voxel-based notation,then the individual voxels can contain weightings. The probabilities canalso flow into this weighting and thus amplify the advantage of suchweightings. Further explanations about how weightings work and whatadvantages they bring for the 3D scanning can be found, for example, inUS 2015/0024336 A1.

According to another modification of the invention, the images arerecorded while the structures are sequentially illuminated with one ofat least two, preferably four, types of light. This produces groups ofimages that each depict intraoral structures illuminated in a differenttype of light. This allows information to be gleaned that can bevaluable for both the generation of virtual three-dimensional surfacegeometries, and also for the segmenting.

A first possibility and preferred implementation of the invention is forexample that the images are recorded while the structures aresequentially illuminated with one of three types of light and the threetypes of light are light that is projected in different colors,preferably red, green, or blue. It is thus possible to glean colorinformation even with a monochrome sensor.

It is thus possible, for example in another preferred implementation ofthe invention, for the segmenting to be carried out by the neuralnetwork based on a reflected light intensity as a function of theprojected color. For this purpose, use can be made of the fact thatteeth, because of their opalescent properties, reflect blue light, whichpenetrates less deeply, differently than for example red light, or thefact that gums in red light appear different than they do in greenlight.

In another preferred implementation of the invention, one type of lightis projected onto the intraoral structures in a structured way,particularly in the form of a pattern. In this case, structured lightcan be particularly helpful in obtaining particularly precise geometricsurface information. For example, structured light can exist in the formof regular or irregular patterns, which are projected onto the intraoralstructures, for example as uniform stripes or randomly distributedpoints. Structured light, however, can also be laser light, for example,with the speckles that strike the illuminated objects being used toimprove the geometrical surface information.

In another quite particularly preferred implementation of the invention,a region of the virtual three-dimensional surface geometry is updated bymeans of data from pixels, which correspond to the region and are fromat least one two-dimensional image, and a detection is carried out as tohow often the pixels have a particular label. It is thus possible withregard to every region to glean information as to how often the regionhas been updated with a particular label.

From this information, it is possible to obtain various additionalinformation that makes it possible to glean information about the degreeof probability that the segmenting has occurred correctly or that simplyenables a more robust scanning procedure.

Other preferred implementations of the invention are the subject of theremaining dependent claims.

BRIEF DESCRIPTION OF THE DRAWINGS

A preferred implementation example of the invention is described ingreater detail below based on the drawings. In the drawings:

FIG. 1 shows a diagramed depiction of an exemplary method according tothe invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the depiction in FIG. 1, first a series of two-dimensional images 2is made of intraoral structures 1. Depending on the method with whichthe two-dimensional images 2 are converted into three-dimensionalsurface information, it is theoretically also possible for a singletwo-dimensional image 2, which is converted into three-dimensionalsurface information, to profit from the advantages of the method. In thecontext of the example shown, however—without limiting the invention tothis—there is assumed to be a series of at least two successive images2.

The two-dimensional images 2 are then partitioned pixel-by-pixel intosegments 3, 4, 5, 6. There are different types of segments, with eachtype of segment corresponding to one type of intraoral structure. In thesimplest implementation, it is possible for example to distinguishbetween the types “teeth and tooth-like intraoral structures” and“remainder.” In enhanced implementations, it is also possible, forexample, to consider “gums,” “soft tissue,” “transition between gums andteeth,” and the like as separate types of intraoral structures and ascorresponding types of segments 3, 4, 5, 6.

In a preferred implementation of the invention, the individual teeth andtooth-like objects are each considered to be a separate segment with aseparate label. In this connection, no notice is taken of which toothoccupies which position in the mount; instead, it is simply assumed thattwo tooth segments, which are for example separated by a transitionsegment or a remainder segment, each also correspond to two separateteeth or tooth-like structures.

Tooth-like structures or objects as defined by the invention are allintraoral structures that are positioned in the region of the teeth orin lieu of the teeth. Non-exhaustive examples of these in this caseinclude crowns, implants, and inlays as possible tooth-like objects orstructures, which are not themselves teeth, but are treated as teeth forthe method according to the invention.

After the two-dimensional images 2 have been partitioned into two(teeth, remainder) or more segments 3, 4, 5, 6 (teeth, gums, softtissue, transitions, etc.), a selection is carried out as to thesegments with which additional calculations are to be carried out in thesubsequent steps.

The segments 5, 6 with which no other calculations are to be carried outcan be discarded, but they can also remain stored without furtherprocessing and used as needed for other processes.

The selected segments 7 can then be used to generate 2.5D surfaces 8. Inthis procedure, it is important that the respective surface regions ofthe 2.5D surfaces 8 continue to correspond to the previously selectedsegments 7 and this, in accordance with the labels assigned in thetwo-dimensional images 2.

The method of converting the two-dimensional images 2 to the 2.5Dsurfaces 8 can be freely selected by the person skilled in the art andessentially depends on the selected method with which thetwo-dimensional images 2 were generated. If the images were generatedstereometrically, then it is possible to determine disparities betweenimage pairs. If the images 2 were generated with simultaneous projectionof a known pattern, then in each individual image, the surface structurecan be determined based on the distortion of the pattern. These twopossibilities are mentioned here merely as examples. The person skilledin the art can freely choose from among the available methods that aresufficiently known and described.

It is advantageous that in this step, it is already possible to achievesavings in computing resources since only the pixels in thetwo-dimensional images that also have the corresponding labels are usedfor the conversion.

As is also the case with the known methods in the prior art, the nextstep of the method consists of bringing the individual 2.5D surfacesinto relation to one another and/or to an already existing virtualthree-dimensional surface geometry and combining or adding them. Thiscan be done using the ICP algorithm, for example, which is sufficientlyknown to the person skilled in the art. Extensive explanations for thestep known from the prior art can be found, for example, in EP 2 672 461A1.

What is new relative to the prior art is the special possible actions aswell as the advantages that ensue from the 2.5D surfaces, which arepartitioned into segments, since in this step, a search is carried outfor surfaces, which at least partially, can be essentially mapped ontoone another. In this case, “partially” means that the individualsurfaces do not have to completely overlap. The fact that the surfacescan be essentially mapped onto one another includes the fact that in thecontext of measurement errors, the surfaces generally cannot be mappedonto one another with 100% accuracy.

The adaptation of the method according to the invention is accompaniedby two significant advantages or improvements relative to the prior art.On the one hand, it is not necessary to fit in the surfaces that areneither teeth nor gums, for example an inadvertently recorded tongue orsoft tissue beyond the gums since no 2.5D surfaces are generated forthese. On the other hand, erroneous matchings or imaging can beminimized because of the limitation that only surfaces that representteeth are mapped onto surfaces that represent teeth and those surfacesthat represent gums are mapped onto surfaces that represent gums.

Preferably, for the virtual three-dimensional surface geometries 9, italso remains noted which regions (and now surfaces) were labeled aswhich segments in the original two-dimensional images. According toanother modification of the invention, this information can be used forother optimizations of the virtual three-dimensional surface geometries9.

One such optimization can, for example, be a coloring of the model. Toaccomplish this, the teeth can be colored using white and pink colorsthat usually correspond to intraoral structures. For example,statistical color averages or color values measured directly on theintraoral structures can be used for this.

In another preferred implementation of the invention, not only are theteeth as a whole considered as a separate segment S1, but also eachtooth is considered as a separate segment S1.1, S1.2, S1.3, S1.4. Toachieve this, the system only has to recognize that it is a tooth, butnot which tooth it is (i.e. for example: second incisor, upper left).Since the two-dimensional images are usually generated as a sequencebased on the movement of the intraoral scanner, in two successiveimages, it can be assumed that there is at least an intersection of theimages at least in some regions. A tooth segment in one image is assumedto be the same tooth in a subsequent image until a transition zonebetween two tooth segments is detected. Starting from this point, it isthen assumed that it is a different tooth.

A thus enhanced method therefore only detects the “first tooth seen,”“second tooth seen,” etc., but not whether it is, for example, anincisor or whether it is positioned in the upper or lower jaw.

Virtual three-dimensional surface geometries based on such an enhancedsegmenting can be used, for example, to mask out individual teeth in themodel or for example also to individually move teeth as part oforthodontic simulations.

REFERENCE NUMERAL LIST

-   -   1 intraoral structures    -   2 two-dimensional images    -   3 segment S1 (tooth)    -   4 segment S2 (gums)    -   5 segment S3 (transition)    -   6 segment S4 (remainder, soft tissue)    -   7 selected segments (3, 4)    -   8 2.5D surfaces    -   9 virtual three-dimensional surface geometries

The invention claimed is:
 1. A computer-implemented method for detectingthree-dimensional surface geometries of real intraoral structures,comprising: starting with a series of two-dimensional images of anintraoral structure, converting the two-dimensional images of theintraoral structures into a virtual three-dimensional surface geometryof the intraoral structure, wherein said converting step comprisespixel-by-pixel partitioning of the pixels of the two-dimensional imagesinto at least one segment, each segment being a two-dimensional imagesegment, assigning each segment a label, wherein, each segmentcorresponds to at least one type of real intraoral structure, there areat least two types of segments, each type of segment has a differentlabel, and at least one of the labels represents teeth and tooth-likestructures, and selecting at least the segments of at least one label togenerate the virtual three-dimensional surface geometry of the intraoralstructure, wherein the types of intraoral structures having differentlabels include at least one of the following types: soft tissue,including tongue and cheeks, hard tissue, including gums, tooth-likestructures, including teeth or dentures that are modeled on teeth,artificial structures, transitions between individual types ofstructures including separating teeth, and remainders separating twoteeth, and wherein, for each recorded tooth, a separate label isassigned, labels are assigned to the transitions and the remaindersbetween the recorded teeth, the labels assigned to the transitions andthe remainders between the recorded teeth and to each recorded tooth areselected to generate the virtual three-dimensional surface geometry ofthe intraoral structure, the virtual three-dimensional surface geometryshowing each recorded tooth, the method using the labels assigned to thetransitions and the remainders between the recorded teeth to positionthe recorded teeth with respect to each other in the virtualthree-dimensional surface geometry.
 2. The computer-implemented methodaccording to claim 1, wherein the segmenting takes place by means of anartificial intelligence.
 3. The computer-implemented method according toclaim 1, wherein for each recorded tooth, a separate label is assigned,and the labels assigned to each recorded tooth are selected to generatethe virtual three-dimensional surface geometry of the intraoralstructure, the virtual three-dimensional surface geometry showing eachrecorded tooth.
 4. The computer-implemented method according to claim 1,wherein the at least one label is selected and only the segments withthe selected label(s) are used to generate the virtual three-dimensionalsurface geometry of the intraoral structure.
 5. The computer-implementedmethod according to claim 1, wherein the labels are also retained in thegenerated virtual three-dimensional surface geometry.
 6. Thecomputer-implemented method according to claim 5, wherein the retainedlabels are used for optimizations including a coloring of the model. 7.The computer-implemented method according to claim 1, wherein inaddition to the type of intraoral structure, the labels also include aprobability that indicates how probable it is that a correct label hasbeen assigned.
 8. The computer-implemented method according to claim 7,wherein the probability of the label flows into a weighting of thevirtual three-dimensional surface geometry.
 9. The computer-implementedmethod according to claim 1, wherein the images are recorded while thestructures are sequentially illuminated with one of at least two typesof light.
 10. The computer-implemented method according to claim 9,wherein the images are recorded while the structures are sequentiallyilluminated with one of three types of light and the three types oflight are light that is projected in different colors.
 11. Thecomputer-implemented method according to claim 9, wherein the segmentingis carried out by a neural network based on a reflected light intensityas a function of a projected color.
 12. The computer-implemented methodaccording to claim 9, wherein one type of light is projected onto theintraoral structures in a structured way, including in the form of apattern.
 13. The computer-implemented method according to claim 1,wherein the images are taken using a monochrome sensor.
 14. Thecomputer-implemented method according to claim 1, wherein a region ofthe virtual three-dimensional surface geometry is updated by means ofdata from pixels, which correspond to the region and are from at leastone two-dimensional image, and a detection is carried out as to howoften the pixels have a particular label.
 15. The computer-implementedmethod according to claim 14, wherein the detection of the label iscarried out for each label.
 16. The computer-implemented methodaccording to claim 1, wherein the segmenting takes place by means of anartificial intelligence and the artificial intelligence is a neuralnetwork.
 17. The computer-implemented method according to claim 1,wherein the segmenting takes place by means of an artificialintelligence and the artificial intelligence is a convolutional neuralnetwork.
 18. The computer-implemented method according to claim 1,wherein plural labels are selected including the at least one labelrepresenting teeth and tooth-like structures and only the segments withthe selected labels are used to generate the virtual three-dimensionalsurface geometry of the intraoral structure, the intraoral structurebeing the teeth or the tooth-like structure.
 19. A computer-implementedmethod for detecting three-dimensional surface geometries of realintraoral structures, comprising: starting with a series oftwo-dimensional images of an intraoral structure, converting thetwo-dimensional images of the intraoral structure into a virtualthree-dimensional surface geometry of the intraoral structure, whereinsaid converting step comprises pixel-by-pixel partitioning of the pixelsof the two-dimensional images into at least one segment, each segmentbeing a two-dimensional image segment, assigning each segment a label,wherein, each segment corresponds to at least one type of real intraoralstructure, there are at least two types of segments, each type ofsegment has a different label, and at least one of the labels representsteeth and tooth-like structures, and selecting at least the segments ofat least one label to generate the virtual three-dimensional surfacegeometry of the intraoral structure, wherein, segments having differentlabels include: tooth-like structures positioned in a region of theteeth or in lieu of the teeth, the tooth-like structures includingcrowns, implants, and inlays, transitions between individual teeth andthe tooth-like structures, remainders separating teeth and thetooth-like structures, for each recorded tooth, a separate label isassigned, for each recorded tooth-like structure, a separate label isassigned, labels are assigned to the transitions and the remainders, thelabels assigned to the transitions and the remainders, to each recordedtooth, and to each recorded tooth-like structure are selected togenerate the virtual three-dimensional surface geometry of the intraoralstructure, the virtual three-dimensional surface geometry showingrecorded teeth and recorded tooth-like structures, the method using thelabels assigned to the transitions and the remainders to position therecorded teeth and the record tooth-like structures with respect to eachother in the virtual three-dimensional surface geometry.